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Prof.  Kurt  VanLehn  &  Dr.  Joel  Martin 
Learning  Research  and  Development  Center 
University  of  Pittsburgh 
Pittsburgh,  PA  15260 


Dear  Susan, 

This  letter  is  the  seventh  quarterly  progress  report  for  grant  N00014-91-J-1532,  entitled  “On-Line 
Assessment  of  Expertise.”  It  covers  the  period  April  1,  1993  to  June  30,  1993. 

There  were  two  main  objectives  during  this  report  period.  The  first  was  to  extend  Olae’s  physics 
rule  base  or  cognitive  domain  model  to  include  incorrect  rules  and  correct  but  unusual  rules.  The 
second  was  to  determine  whether  Olae  was  able  to  match  fine-grained  assessments  performed  by  a 
person. 

Although  Olae  can  model  many  correct  and  incorrect  solution  methods,  the  students  in  our  first 
sample  occasionally  acted  in  unexpected  ways.  Before  this  reporting  period,  Olae’s  and  Cascade’s 
physics  rule  base  was  sufficient  to  solve  many  problems  in  several  chapters  of  introductory  physics 
and  contained  some  common  buggy  rules.  To  account  for  the  unexpected  methods,  we  added  rides 
sufficient  to  produce  the  behavior  exhibited  by  the  students.  Of  course,  because  we  used  this  sample 
to  augment  the  rule  base,  we  will  collect  another  sample  to  evaluate  Olae. 

The  second  objective  of  this  period  was  to  evaluate  Olae’s  performance.  We  have  compared  Olae’s 
assessment  for  two  complex  physics  problems  to  assessments  performed  by  a  human  from  verbal 
protocols.  These  protocols  were  collected  and  aUhlyzed  before  Olae  was  constructed  and  were  not 
used  in  the  development  of  Olae.  In  all  cases  where  the  human  assessor  determined  that  the  student 
knew  a  rule,  Olae  had  assigned  a  probability  of  greater  than  0.85  to  that  rule.  Similarly,  in  all  cases 
where  the  human  assessor  determined  that  the  student  did  not  know  a  rule,  Olae  had  assigned  a 
probability  of  less  than  0.15.  In  other  words,  Olae  is  ais  accurate  as  the  human  assessor.  In  the 
future,  we  will  extend  this  analysis  to  twenty  problems  and  with  five  subjects. 
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LRDC  Senior  Scientist 

& 

Joel  Martin 

LRDC  Research  Scientist 

cc:  ONR  Resident  Representative  N66005,  OSU  Research  Center 
Director,  Naval  Research  Laboratory,  Code  2627 
Defense  Technical  Information  Center 
University  of  Pittsburgh  Contracts  Office 
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